Note that all the claims currently pending in this application, including those not presently 
being amended, have been reproduced below for the Examinees convenience. 



1. (Amended) A simulator apparatus with which an operator plays a 
simulation with virtual object(s) in mixed realijfy space including a virtual space and real 
space, said simulator comprising: 

viewpoint-detection means fof detecting the location/posture of a viewpoint 
of the operator; 

geometric information acquisition means for acquiring geometric 
information of real object(s); 

recognition means for recognizing a current relative relationship between 
the virtual object(s) and real object(s); 

a rule memory for storing rules for controlling the action of the virtual 

object(s); 

computation means for determining the next action of the virtual object(s) in 
accordance with the rules stored in said rule memory and in correspondence with the 
location/posture of the real object(s), and computing the location/posture of the virtual 

nd 

generating at least one image of the virtual object(s) 
the virtual object(s) after the determined action and 



object(s) after the determined action;- £.1 

presentation means for 
on the basis of the location/posture of 



the location/posture of the viewpoint oosition of the operator, and for representing the 



mixed reality space to the operator by 



on the operator's view of the real space. 



superimposing the image(s) of the virtual object(s) 
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^^Cj 2. (Amended) The apparatus according to claim 1, wherein said presentation 
means further comprises: 

image-capturing means for capturing real space images of said operator's 
view of real space images of said player's view of the real space; 

image generation means for generating mixed reality images representing 
the mixed reality space by superimposing orjoverlaying said image(s) of the virtual 
object(s) on said real space images; and 

a video see-through type display means that the operator wears wherein said 
mixed reality images are displayed. 



3. (Amended) The apparatufs 
means further comprises an optical see 
wherein said virtual object image(s) are di 



according to claim 1, wherein said presentation 
though type display means that the operator wears 
played. 



4. (Amended) The apparatus according to claim 1, further comprising, 
status detecting means for detecting status of the operator; 
wherein said computation means determines a next action of the virtual 
object in accordance with the rule stored inlsaid rule memory and in correspondence with 
the location/posture of the real object and/or the status of the operator, and computing a 
location/posture of the virtual object after thk determined action. 



5. (Amended) The appara^ 
relative relationship includes a layoi^^ationship^betv^een the virtual object and real 
object at the current time in the mfxed realjty 'space. 



6. (Amended) T\e apparatus according to claim 1, wherein the current 
relative relationship includes a behavior of the real object with respect to the virtual object 
at the cdrrent time in the mixed reality space. 
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7. (Amended) The apparatu^ according to claim 1, wherein the real object 



includes the operator himself or herself, apd said recognition means recognizes a current 
relative relationship between the virtual abject and the operator. 




8. (Amended) The apparatus acgordmg o claim 1, wherein the real object 
'includes a plurality of operators wJioiSperate said simulator apparatus, and the plurality of 
operators share a singlenrixed reality space. 



vlm! 9 * ( Amended ) The apparatus according to claim 1, wherein the real object 

is an object which is fixed in position, and 

said geometric information ac juisition means comprises: 

a predetermined memory for \ re-storing location information and shape 

information of the real object; and 

means for reading out the location information and shape information of the 

real object from said predetermined memory as needed. 
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10. (Unamended) The apparatus according to claim 1, wherein the real 



object is an object which is movable but does not 



deform, and 



said geometric information acquisi ion means comprises: 

a predetermined memory f( r pre-storing shape information of the 
real object; 

a location/posture sensor f§r detecting a location/posture of the real 
object; and 

means for setting a region he real object is expected to occupy in the 
mixed real space in accordance v|th the detected location/posture of the real 
object. 

11. (Amended) The apparatus according to claim 1, wherein the real object 
is an operator, and 

said geometric information acquisition 
a sensor for detecting a . 

and 



means for setting a region 
approximates the operator in the 



means comprises: 
lbcation/posture of a head of the operator; 



having a fixed, known shape that 
mixed reality space in accordance with the 



detected location/posture of the operator's head. 




12. (Amended) The apparatus.according to claim 1, wherein when the 
simulation is a battle simulatioi^itli^^^al object, an objective is to decrease an 
expected score of the operator. 
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13. (Amended) The apparatus according to claim 1, wherein when the 
simulation is a cooperative simulation with the virtual /object, an objective is to increase an 
expected score of the simulation. 



14. (Amended) The apparatus according to claim 1, wherein one of the 

/ 

rules stored in said rule memory controls the action ^Ohe"^ual\)bject on the basis of an 
objective of the simulation and a relative relatiohshijp between the virtual object and real 
object. 



15. (Amended) The apparatus according to claim 1, wherein one of the 



rules stored in said rule memory expresses the action of the virtual^object as an action 
pattern with a predetermined aim for achieving aii^bjecfive of the simulation. 



16. (Amended) The apparatus according to claim 15, wherein the pattern 
has a path disadvantageous to the operator in consideration of a layout relationship between 
the virtual object and real object. 
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qC, f 17. (Amended) The apparatus According to claim 11, wherein when the 



operator is one of the real objects, an output/from said viewpoint detection means for 
detecting the location/posture of the viewpoint of the operator is also used as information 
which is to be acquired by said geometric infomation acquisition means and pertains to a 
location and shape of the operator. 



\ 



18. (Amended) The apparatus/according to claim 1, wherein said viewpoint 
detection means detects a location/posture or the head of the operator, and 

said apparatus further comprises detection means for detecting a 
location/posture of a hand of the operator; and 

means for recognizing a relative location of the hand of the operator with 
respect to the head as a command on the basis of an output from said detection means. 



19. (Unamended) The 



presentation means comprises: 

means for aligning the 
location/posture of the virtual object t 



apparatus according to claim 1, wherein said 



location/posture of the real object to the 
fter movement; 



correspondence with an occlusion rel; 



means for generating ai image of the virtual object after alignment in 



ionship; and 



a head-mounted display device. 



20. (Amended) An image processing method for a simulator apparatus with 
which an operator plays a game with virtual object(s) in a mixed reality space comprising: 
viewpoint detection step tpr detecting the location/posture of a viewpoint of 

the operator; 

geometric information acquisition step for acquiring geometric information 
of real object(s); 

recognition step for recognizing a current relative relationship between the 
virtual object(s) and real object(s); 



computation step for determining the next action of the virtual.0l5ject(s) in 



accordance with the rules stored in a rule memory, whiph storesjrutes'for controlling the 
action of the virtual object(s), and in correspondence \^i^the1ocation/posture of the real 
object(s), and computing the location/posture of Jhe'vjlrtual object(s) after the determined 
action; and 

presentation step-for generating at ledst one image of the virtual object(s) on 
the basis of the location/posture of the virtual objects) after the determined action and the 
location/posture'oft^viewpoint position of the operator, and for representing the mixed 
\ reality spaceto the operator by superimposing the|image(s) of virtual object(s) on the 
op^ratOT's view of the real space. 



(L 21. (Amended) The method acc6rding to claim 20, wherein the operator 



fears a video see-through type display, and said presentation step further comprising, 



image-capturing step for capturing real space images<of said operator's view 



of the real space; 

image generation step for generating mixed reality images representing of 



the mixed reality space by superimposing or overlaying said image(s) of virtual object(s) 



on said real space images and for displaying s 



/ 



d mixed reality images on the display. 



/ 



22. (Amended) The method according claim 20, wherein the operator 



wears an optical see-through type display and said presentation step representing the mixed 



/ 



reality space to the operator by displaying the image(s) of virtual object(s) on the display, 
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23. (Amended) The method according to claim 20, furthe^ 
status detecting step for detecting status of the^erator; 
wherein said computation step determines^ next action of the virtual object 
in accordance with the rule stored in said mleinemoi^ ind in correspondence with the 
location/posture of the real obipet^ndJor^ht status of the operator, and computing a 
location/posture otihcvivU^l object after the determined action. 



24. (Amended) The method according to claim 20, wherein the recogniti&n 
step recognizes the current relative relationship including a layout relationship^etween the 
virtual object and real object at the current time in the mixed reality^ace. 



25. (Amended) The method according to claim 20, wherein the recognition 
step recognizes the current relative relationship including a behavior of the real object with 
respect to the virtual object at the currerfftoie in the mixed reality/Space! 



26. (Amended) The method according to claim 20, wherein the recognition 
step includes the stepM recognizing a current relative relationship between the virtual 
object and the operator, and the real object includes the operator himself or herself. 




27. (Amended) The method accordinglo claim 20, wherein a plurality of 
operators share a single mixed reality space-and the real object(s) used in the computation 
step includes a plurality of operators who operate the apparatus. 




(tot 



28. (Amended) The method according to claim 20, wherein the real object 
is an object which is fixed in position, and 

the geometric information acquisition step includes the steps / 
pre-storing location information and shape information cjf the real object in 

a predetermined memory; and 

reading out the location information and shape information of the real object 

from the predetermined memory as needed. 

29. (Unamended) The method according to claim 20, wherein the real 
object is an object which is movable but does not deform, and 

the geometric information acquisition step includes the steps of: 

pre-storing shap/information of the real object in a predetermined 
memory; 

detecting a location/posture of the real object by a location/posture 
sensor; and^ 

^/^etting a region the real object is expected to occupy in the mixed 
real space in accordance wit h the detected location/posture of the real 
'object. 



30. (Amended) The method according to claim^20f wherein the real object 
is an operator, and 

the geometric informationkcquisition step includes^e steps of: 
detecting a locati^n^o^re^of the head ofjhe operator; and 
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setting a region having a fixed, known shape that approximates the^erator 
in the mixed reality space in accordance with the detected location/posture^ 

31. (Amended) The method according to claim 20, wherein when the 
simulation is a battle simulation with the virtual object, an objective used in said 
computation step is to decrease an expected score of the operator., 

32. (Amended) The method according to claim 20, wherein when the 
simulation is a cooperative simulation with the virtual objeetran objective used in said 
computation step is to increase an expected score of the operator. 

/ 
/ , 

33. (Amended) The method/according to claim 20, wherein one of the 

/ / 

rules controls the action of the virtual object on the basis of an objective of the simulation 

/ / 

and a relative relationship between the virtual object and real object. 




34. (Amended) /The method according to claim..20rwherein one of the 

A 

rules stored in the rule memory expresses the actjpn-fiT the virtual object as an action 
pattern with a predetermined aim for achieving the objective of the simulation. 



35. (^mended) The method according to claim 34, wherein said 
computation step determines an action using a pattern having a path disadvantageous to the 

/ 

operator in consideration of a layout relationship between the virtual object and real object. 



- 11 - 



36. (Amended) The method accord^gl^claim 3Q^wherein when the 
operator is one of real objects, said geometric informatiofi acquisition step uses an output 
from said viewpoint detection step of detectfi^ of the 



operator that pertains to a location^and shape of the operator. 



37. (Amended) The method according to claim 20, wherein theviewpoint 



detection step includes the step of detecting a location/posture of the head of the operator , 




and 



said method further comprises the detection step of detecting a 

/ 

location/posture of a hand of the operator; and / 

the step of recognizing a relative location of the hand of the operator with 
respect to the head as a command on.tliebasis of an output in the detection step. 



38. (Unamended) The method according to claim 20, wherein the 
presentation step includes the steps of: 

aligning the location/posture of the real object to the location/posture of the 
virtual objecUifter movement; and 



/ 



/ 



/ 



generating an image of the virtual object after alignment in correspondence 



with an occlusion relationship. 



^ Jb ^| 39 - ( Amended ) A storage meflium which stores a program of an image 
processing method for a simulator apparatus with which an operator plays a simulation 
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